Skip to content

fix: add cache-aware Claude session routing#7

Draft
Timo972 wants to merge 17 commits into
VictorMinemu:mainfrom
Timo972:fix/intelligent-session-routing
Draft

fix: add cache-aware Claude session routing#7
Timo972 wants to merge 17 commits into
VictorMinemu:mainfrom
Timo972:fix/intelligent-session-routing

Conversation

@Timo972

@Timo972 Timo972 commented Jul 20, 2026

Copy link
Copy Markdown

Summary

  • keep each valid Claude Code session on one Anthropic subscription account for prompt-cache locality
  • assign new sessions by in-flight load, active bindings, rate-limit headroom, and a rotating tie-break
  • track idempotent request leases through full SSE or JSON response completion and client disconnects
  • invalidate affinity safely after 401, 429, and 529 without retrying or rewriting the current response
  • preserve upstream SSE bytes exactly; never synthesize message_stop
  • harden refresh-token ownership and account deletion across remove/re-add and concurrent refresh races
  • expose aggregate in-flight and active-session counts in health/account views

Root cause

CC-Router selected an Anthropic account independently for every /v1 request. With multiple concurrent Claude Code sessions, request-level round robin moved one conversation across account-specific prompt caches and ignored long-running in-flight streams. This diluted cache reuse and could concentrate concurrent upstream work in ways that surfaced as a 300-second stalled-stream error. The timeout was the visible trigger, not the underlying routing defect.

User impact

Claude Code sessions now remain sticky while their account is usable. New sessions distribute across available capacity, and account-specific failures affect only the next retry. Response status, headers, body bytes, event order, and timing remain under native http-proxy-middleware passthrough.

Session mappings are memory-only, expire after one hour, are capped at 10,000 entries, and are never logged or persisted.

Validation

  • npm test — 36 files and 287 tests passed
  • npm run lint
  • npm run build
  • byte-exact concurrent production-stack SSE coverage
  • regressions for lease cleanup, duplicate session headers, stale response generations, account ID reuse, cooldown overlap, refresh/delete ownership, persistence retry, and client disconnect during refresh

Note: tests still emit the existing Node DEP0060 warning from an upstream dependency using util._extend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant